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ABSTRACT 

The traditional relialility co^^ficiert erd standard 
error o/f measurement are not adeguate neasuies c\ leliatili ty for 
^1:ests used to make pass/fail decisiqijs. insiier irg ^±h^ iEfcr-tant 
reliability questions reguires^ ^^sti-ttaticn of* the joint distribution 
of true and observed scores. Lotd»£ •»£ethcd 20" estittates tii^ * 
distribution without the deficiencies of ctleE n'ethcds. New output 
"formats condense the estimated di£tribi:tici into readily usable 
information^j including a 2 x 2 contingercy. tabl^e, ccnditicnal ^' 
true-score distributions^ and an index of decisicn-Eakirg efficiency. 
Examples are' appended. (Author/CTH) ' ^' 




^ ' Reproductipns supplied by.^DRS arefthe best th^t can be made * 

from the original document. ;* 
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Reliability of tests used to make pass/fail decisions: 
right questions. 

SAMUEL A. -LIVINGSTON • ' - 

Center for Occupational and Professional Assessment • 
Educational Testing Service ^ 



Answering the 



/ 



The traditional reliability coefficient and standard error 
of measurement are not adequate measures of ' reliability for tests , 
used to make pas^f^il decisions.' Answering the important reliab'ility 
questioos' requires estimation of the joint "distribution of true ajid 
observed scores. Lord's "Method 20" esfimaf^s this distribution 
without the deficiencies of otijer methods. New output formats 
ccftidense the estimated distribution into readily usable information, 
including a 2 x 2 contingency table, conditional true-score distri- 
butions^ and an index of decision-making efficiency,. 
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Reliability o^- tests used to make' 
pass/ fail decisions: ^ Answering^ • 
" the right questions 



- ' Samuel A. Livingston 

,1 Center for Occupational and Proves sj-onal A^^essment 

Educi^tional Testing Serviee_ 

'V- ^ . ■ 

A§ the title- implies ,^ this paper is about reliability - the relia- 
bility^ oi. a particular type of test. The tests i am concerned about 
are^,tests, that have the follt)wing three characteristics: First, the test 
is made up of items scored as ri^ht or wrong.^ Second, ^the test score 

will be Used as the basis for a pass/fail decision. Third, the variable 

/ ' . • ^ ' \ 

measured by the test is continuous; a person's true score can be anywhere 

. ^' 1,** \' 

on the continuum; Notice that -I am no€ talking about a situation' in 

which the test 'scores are supposed j::d reflect a true dichotomy. \ In the ^ 

•kind of te^sting I am concerned about, , the terms "mastery" and "ndn-mastery" 

' are somewhat misleading; it' makes more ^ense to talk about "minimally 

acc'eptable performance". It is easy to think of exataiples.of tests\of . ♦ 

this type: A final exam for a course, a licensing exam for a profession, 

a screening test for a remedial program, and- so on. 

^ The traditional measures of test/ reliabilit;^ are the reliability 

coefficient and the standard error of measuremejit:^ What do thea^e tr^adi- 

^ tional measures really tell us? The reliability coefficient answers! the . ' 

question: "What is the estimated correlation between two equivalent forms 

^v:-- ; '■■ ■ - V ■ 

. ^ of thevtes.t, in the, group of^persons taking' t)ie' test?" Xhe standard \ 
ertor 'erf measurement answers the question: "What is the average size of 
: ■' , th,e errors of ffl^surement, in, the grdup of persons, taking the test?" But" 

v /■ . • " ■ ■ . m ■■ ■ Ay 




these are hot the important questions. What neither the reliability 
coefficient nor the overall standard error of measurement telPs us is how 
much confidence we can Tiave iiiHthe pass/fai;L decisions we are making on the 

> ^ c 

basis of the test, » . 

What kind of questio'ijs should we be asking? 'ihe important questions 
of pliability have to do with pass/fail decisions and the relation- ' ' 
ship Qf theae deoisions to the true scores of the persons taking the t^est. 
(By "true score", I mean the person' s average^^sc ore ove^ the universe, of ^ 
all possr^le tests.) Some of the questions .we shduld be, asking are th^e: 
^ . ' 1. Of thosQ persons who passed the test, how many would have -Xy f 
passed if the test had been perfectly reliable? Of tho^e who 

failed, how many wpuld have failed if ther'test had been perfectly ' 

^ / 
reliable? ' » 

> . • ; • . 

2. What is the distrib-ution of true scores in the grg^up of persona 

^< ■ 

who pasaed the test? What is the distribution of true scjores ^ 
in the group of persons who failed the te^t? ' ' ^. 

3, What is the decision-making efficiency of the test, is cdipared ' 
with that of a perfectly- reliable test? - ' " ^ 



To .answer questions like these, we peed to estimate th$ joint ' * ' ' 

distribution of true scoyes '^ and observed scores. This^s not' a* simple 

problem. Fred Lord worked on it for several y6ars, and-h^ de.velbped *. * 

several methods of estimating this joint dfstributuon. the 'mo»tr advaniied 

* - * . * ^ * *^ 

of these was published in 1969. It is callecl ^simply "Metjhod ^SO"^ ,r^^e ' '."^ 

purpose of this paper is to describe briefly some o^ the- , characteristics 

of this procedure and to show how it can be used to arisWer the kinds* of 

reliabitity' questions we should be asking.* * \ ^ \ / 



' Method 20 assumes that the conditional dist!fibution of observed, 
scores, for persons with* a given true score, is a "compound binomial 
distribution".'' This assumption is less restrictive than the ^ssumption 
of a binomial distribution. Strictly speaking, the binomial ^assumption 

■ / 

would be appropriate only if all test items were equally dirfficult £r if 
each person taking- the te&t received a separate random san/pLe of items. 
(Actually, Method 20 can also be used with the binomial afssumption, and 
the results turn out not to be very different. 

Method 20 is somewhat unusual in that -the mathematical model for 
the true-j&core distribution does not have a fixed number of parameters. 
^In fact, the ni&ber of parameters in the model is^ limited only by the 

number of items on the test. This feature makes the*model very flexible. 

' ~~ — ^ " . • ' ' 

Even^if the distribution of true scores happens^ to h*ave some unusual \ 



shape, ^-Method 20 can eatimate it. Othey methods^ for ex'ample those 
based on the beta distribution - cannot estimate a bimodal disttibiltion 
of true score^. ' c • 

Me^thod 2Q does have some limitatidns. The t^st must be unsp.eeded , 
and there mu^t bfe no correction for guessing. Ideally, . e;^ery person 
Shmild have a chance to ansi^^er ever/ item, and the person's score must* be 
simply the number of items answered correctly. The only lipit o^ the 
length pf the test is a practical one; the existing comput^T.^ program will 
not accept a test o;t more than 100 items. Also, a very short te$t\^limits 
the complexity of the estimatec^ :rue-scdre distribution, fiat e^^en as few 

' . . . ■ ^ ' .-. . ■ '■ \ - ; 

as six items will allow for a bimodal distribution with unequal modes. 

, The required input 0r Method ^0 consists of- the oblserved score 
distribution and the variance of item dificulty' values. The catch is 
tHat this. Jjifarmation has to come from a large sample - p'teferably 1000* 
or more persons* Howeverpthe^ input does not have to come from a test' * 



9 

-4- 



of the same length as the one for which we want the 'estimates/^ That 

is^, we .can 'use data from a 20-item test t(^Hget an estimated distribution 

. . / ^ * ' ' ' \ ' 

for a 50-itiem test, or a lO-item test, or a test of any length we want ' 

to specify. This feature^is -especially' usejul in/deciding how- loiig a^ * 
test needs**^G be. ' ' 'i ^ * / 

. • ' . . ' / . , 

Since I am neith^er a mathematician nor a -computer programmer, \i 
\ wonjt attempt to explain either the mathematical solution. or the computer 
^ progracuni'rrg of Methoi\20.-' Both were published' in 1969. The mathem"ktical 
solution is m an article by I^ord in P'sychometrika ; the computer prdg^ram 
IS m an article by Wj^ngersky arid others in Educat^ron^l and Psycholdlgic^l 



Measurement* 



What is new lis the collection of outputs formats we at ,ETS have 
recently; developed, and^.the computer program' that produces them». Th* 
formats take th,e ektima^ed ^t^int (distribution of true an<f obse'rved scores 
and. condense it do\ii to a readabl^;_'^isable form, sa that the director ilof 
a "testing program cin use th.e information\as an aid in deciding hdw'lc^ng 
to make; the tests " ^^^^ ^^>' in. d^^f ending tl\e test against a . ; 

claim^ th'at it is, n<St \a reliable basis, for ma^^ng pass/ fail decisions. 



V 



These outg;ut formats are- illustrated, in Ihe handout. The first • 
page shov^i^ contingentTf table, foiled 'ffom the joint ^distribution by 
specifying a pass iCutoff score fc^ir ob-served .scores and a similar 



.1^ 



cutdff jfor true scor^sU The jjercei^lag'es %n eac^i <^eli are estimated^ . 
percentages of the popt^lWtion of eatxamihees*... :fori example^ an estimated 
11% of the exaniinees.wiU pass ^he. -examinati^Ti even though their j true 



scores ^re^below the outpff 



0 ; y I 

*i At' the bottom of th4 Ifirst. page is an . index labeled "(f^cisiori-making^ 



yefficienby,". This indexLik based on- an assumption >f livd&t utility.*, 



Linear ^tility means Vhat'if Jones has 'a true score ten points §lbove the \ 
cutoft, and Smith has* ^^true \scor.e five gpints above the cutoff, TTien 
failing Jones is twic^ as serious an error as .failing Smith. -The efficiency 
index is simp.l^ /the utilit^y of ^electing on observed' scores, divided by 
the utility- of selecting, on\ true .scores. Method 20 makes it possible 
estimate such an'. index. 

of the haLdout- 



the second page 



^shows the estimated distribution of 



vtrue scores .in >the passing group and in t^ie failing group* This 'table 

gives the test user , a good ideja of tlveXamount of separatit)n between the 

, - - ' • \ . » ' 

tvo groups. If the tes/ were perfectly Veliable there would be.no 

.dve^lap between the jt/tue-score distributidns *of the two groups. If the 

test were ''completely unreliable, the two tAie-score distributions would 

be identical • 

The third page' o'f the handout shows the estimated distribution* 
ot true scores for examinees having a specif ied\ observed score. ^ (This 
particular table is for examinees having the loweist passitfg scorej The 
more reliable jthe test, the le.sjs the ^spread - in thi^p distribution. • 

The fourfh page of the IS^doyat sh'ows a series of predicted observ-ed- 
score distributions f6r_ examinees having specified trve .scores. If the /\ 
user does ndt specify the leVels^, the* program. wilL priat* the, distributio^rs 
for true' scores of ;i-0,- ;30, .50-, .70, and\90. These distributions are 
onl^j slightly different from the binomial ^distributions for n = 30 (the 
number of items on the test). ^ ^ " . , 

The next iqhr pa^es of the handout are- the p4me Ss ^the first " ^ . 
four, ^except that the data* from the thirty-item teW has .been used to 



estimate distribtttions for;k sixty-item .test. By cbmparing' the corre- 
spondinc table>^ you can see hoV^uch improvement wodld - resul t front* 
making the* test twice- as long. \The last two pag^s ofiA^he hantiout 
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/ 



are 



. :- 7 



computer-drawn graphs corresponding to ^ the tablesvon pages 6 and 7 of the 
handout. The graphs enable the user/of the program to see the overlap in 
the true-score distributions for, the passing and failing' groui^ and the 4 
spread in the t^rue-^core distribution for examinees with a given observed ^ 
score. ^ (The, numbdrs on the vertical* scale in these graphs do not . correspond 
to those in 6h6 tl|bles, because the true-score, scale^has been divided 
intro intervals of ,01 instead of .05.) * . ^ 

What are the implications 'of this type of reliability estimation? I' 
bplieve that these applications of Cord's work will change^the Ways that 
we in the testing profession think about reliability, at least for 
tests that are intended to be the basis fpr pass/fail decisions. For 
these types ;of tes-fs, I think'we would be justified in saying that the 
traditional .measures of reliability - the relirability cQefficient, and 
the stand^d error of m^surement - ^re now or will soon be obsolete. 
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^andout: page 1 
Printed In IJ^S.A. 



PASS-PAR CONllNCENCV TA3Lf 
NAME OF TfST : 

ORIGlNAt^ATA I . ADMINISTRATION 
THfS 7ABL6 IS FPR A TEST QF 
TRUE-^SCORt CUT-OFF : 

MINIMUM PASSiyG SOPlfr I S 20 OUT OF 30 ITEMS i 67 PERCENT I 





NUMBER OF EXAMINEES =^ 3274 



SHPULO SHOULD " \ ' 
PASS FAIL 



WILL PASS I 55. 9i I U.0% I 
I I I 



WILL FAIL I 8,0X | ?4;3l 1^ 



,1- 



NUMBER OF ITEMS '= 30 



OfcClSION-MAKlNG EFFICIENCY : 0.8I 
DECISION-MAKING EFFICIENCY I-S DEFINED AS 



eXP^TATION(CZ-Z*)\$IGNCX-X*n / 

EXPECTATIONJ (2-2*1 SIGN! Z-2*n / 

WHEfl^e *2* IS TRUE-SCORE CUT-OFF OF 0,67 

X*^IS :OBSERVEf)-StURE CgT-OJFP OF* 20 
I (MINIMUM PASSING^ SCORE' MINUS ONE-HALF "iTEMI 



Handout:' page 2 



CONUIIU'NAL lAUfc-SCUkh OISTK i t*UT IONS FOK PASSING AND FAILING G^ROIJPS^ 



NAKE OF U;ST : 
ORIGINTri^U J 
IMESF DISTRIBUTIONS ARE FOR A TEST UF 30 ^TEMSt 



AfiMINISfKAflON HATE. = 07/01/77 



NUMBER nF EXAMINEES - 3274 NUMBER OF ITEMS > 

WITH A MINIMUM PASSING SCORE OF 20 ITCMS CORRECT 



•30 



. >f?UE«SCORE 
(PROPORT lON-CURRFCT ) 
0.95- 1.00 
O*90- a. 95 
0.05- 0*90 

0.80- "o.as ' 

0.75- 0.^0 

0.70- 0»7S 

0.65- 0.70 

0.60- 0'./»5' 

0.55- t>«oO ^ 

0.50- 0.55 

O'.AS- 0.50 

O.*»0- U.A5 

0.35- O.AO 

a* 30- 0.35( 

C».?5- 0.30 

0.20-. o;<5 
'0.15- O.?0 

O.IO- o.l'".* • 

oCo5- 0.10 
.0.0 r fti*05 , 

TRUE-s/cORt 
• MEAN 
' S.O. 



NUMBER 


PERCENT' N ' 


CUMULATIVE PERCENT 


PASSING 


FAILING 


PASSING 


FAILING 


PASS ir<}G 


FAILING » 


* GROUP 


GROUP 


GROUP 


GROUP 


^ GRJUUP 


GROUP ' 


0. 


0. 


0.0 


0.0' 


100.0 " 


100*0' - w 


20. 


0. 


0.9 


0.0 


100.0 ^ 


, 100. 0 


115.* 


' 0. 


5*2 


0*0 


99.1 


100. o' 


309. 


2. 




0.2 


' • 93.8 


* 100.0 


521. 


' 27.' 


23.8 


?.5 


79.8 


99.8 


570. 


113. 


26.0 


lO.A 


. 56.0 


97.3 


AO 5.- - 


- 236. 


• 18.5 


21*8 


30.0 


86.9 


1A6.^ 


2R5. . 


8.5 


26.3 


1U5 


65.1 


55. 


223.' 


2.5 


20.6 


3*0' 


38.8 


10. 


^.123. 


* 0.5 


11.3 


0.5 


18.2 


1. " 


50. 


0.1 


4.6 


O.I 


6.8 


0. 


15. ' 


0.0 


^ 1.4 


0.0 


2.3 


0., 


4. 


.0.0 


0.3 


0.0 


0.9 


0. • 




0.0 


0.1 


0.0 


0.5 


0. 




0.0 


O.Jl 


0.0 


0.4 


0. 


* 


0.0 


0.1 


* 0.0 


0.3 


0. 




0.0 


J 0.1 


O.tf 


0.3 


0. 


' ^ 1* 


• O.o* 


.0.1 ^ 
0.1 


0.0 


0.2 . 


• > 0. • 




0.0 


0*0 


.0.1 . * 


0. , * 




^o.b 


0.1 


0.0 


0«w 1 > ^ 












t 



\.74 
O.H73 



0.62 
0.080 
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LnNyniPf..L T«.uriVcnKf I;lMKlM|in-N FPK SlUCU OhUJ VCP-UOHE UV^ 



N/Mf OF 1CS1 
ORIGINAL UATA : AnMjNl^lKAlll^ Ta H = i»7/(.l/V7 
IHIS MSIRIHl'linU IS Fflk A Ujr I'F \o I fins 



Klh f XAMlUf 1^ >^nM ITStKVtD- 
(PRfiHlKl U'NHA RHtCl ) 

U.7S- o*eo 

0.>»*i- 0.70 / 
O.S^-- (♦•60 



0*30 



( .10- fj.l! 
^).0S- /♦.lO 

0.00- *o.ri> 

Mf AN 

I 



sunt 

O. 

33. 

?/ . 
6. 
1. 

.(•. 
O. 



?U OUT f> ' 30 



(•.(I 

i 0.(1 

0. 1 

llJ.l 

?6.0 
31.6 

ro.lv 

7.V 

1. r 
a.? 

0.0 

o.c 

. 'o.-o^ 

0»0 

O.l. 

O.U 
• 0.0 

0.0 
\ 0.0 



ITIMS t 67 Rr<?CEN1) 
CUMULATIVE 

^ ^ lOO.ii 
100.(1 
100. O 

30.7 

; ,0 / 

0.0 - , 
'0*0 ^ 

• o.c 

,0*0 / 
0.0 / 

' 0.0 
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Handout u page A 



PREOlCTtO CONDlTlONAt Oli$EKVa»-SCnRL UlSTRlRUTlONS FOR SELECJEO TRUE-^SCORfe.LeVELS 
NAME or TEST z - \ ' 



\ 



ORIGINAL DATA : ADMINISTRATION DATE = 07/01/77 
tHESC WSTklBUTlONS ARE A IGST UF 3o' ITEMS ' 

■>46n-<UMULAT1VC niSTRlf^UTlONS 
TRUE-SCORE I PROPORTION-COHKECT I 
• lOv- .30 .50' ' .70 .90 



ObSCRVLO 
30 - 
29 - 

26 - 

27 - 
76 - 

* 2^ - 
24 - 
23 - 
22 - 

n - 

70 - 
IQ - 
18 - 
H • 
16 " 
1S> *- 
14 - 
13^- 
12 - 
11*^ 
10 - 

7 - 

6, ^. 
5 « 
4 - 

0 - 



-SCORF 

30 

29 

2B 

27 

26 

2b. 

7^ 

23 

22 

?l 

20 

19 

1^. 

17 
(16 

Xb 

1^ 

1? 

12 , 
•11 
10 

8 
7 
6 

7 
1 
0 



NUMBER QF EXAMINEES «= -3274 

^ CUMULATIVE DI^STRIBUTIONS \ 
TRUE-SCORt (PROPORTION-CORRECT) 
•10 .30 . .50 .70 .90 



number\f items' c 



30 



o«.o 

0.0 
0.0 
0.0. 
0.0 
D.O* 
0.0 
0.0 
, 0.0 
0..0 

p.o 

0.0 
0.0 
0.0. 
0.0 
0.0 

0.0: 

0.0 
0.0 
.0.0 
0.0 

p. I 

0. 3 

1. ^ . 

10. «: 

IB. 9 

23.4 
13.0 
3.2 



^ o.e 

0.0 
0.0 
0.0- 
0.0 . 
0.0 
0.0 
0.0 
0.0 
0^0 
^0*0 
. 0.0 
0.0 
0.1 
0.2 
.0.7 
1.8 . 
4.0 
. 7.3 
1U4 
15.1 
16.9 
15.9 
12.4 
7.9 
4.0 

0.5 
0.1 
0.0 
0.0 



0.0 
, 0.0 
0.0 
. 0.0 
0.0 
0.0 
0.0 

b.i 

0.3 
1.0 
2.3 
4.6 
7.9 
1K5 
14.4 
15.6 
14.4 
11.5 
7.9 
4.6 
2.3 
1.0 
0.3 
0.1 
0.0 
0.0 
0«t0 
0.0 
0.0 
0.0 
0.0 



0.0 
0.0 
0.1 
0.5 
U6 
4.0 
7.9* 

12.4 

15*9 

16.9 

15.1 

11.4 
7.3V 
4.0 
. I*'* 
0.7 
0.? 

o.r, 

0.0 
0.0 
0.0 
0.0 
0.0 

;0.0 

o.d 

0.0 
0.0 

<».o 

0.0 / 

0.0 

OtO 



3.2 
13.0 
23.4 
25.5 
1.8.9 
10.^2 
4.1 
1.3 
0.3 
. 0.1 
0.0 
0.0 

^ oip 

. 0.0 
0.0 
0.0 
0.0 
0.0 
0.0 

o.p 

0.0 

0*0 
• 0.0. 

o.e 

0.0 
t).0 
0.0 
0.0 

o.p. 

0.0 
0.0 



100.0 
100.0 
100.0 
100.0 

' 100.0 
100.0 

. 100«0 
100.0 
100.0 
100.0 

lt)d.o 

100.0 
100.0 
-100*0 
5^100.0 
" ICO.O 
100.0 
100.0 
\ 100.0 
100.0 
.100.0 
100.0 
99.9 
99.6 
96.3 
94'. 2 
64.0 
65.0 
^9.6 
16.2 
3.7 



100.0 
100.0 
100.0 
100.0 

ijoa.o 

100.0 
100.0 
100.0^ 
100.0, 
100*0 
100. a 
100.0 
100.0 
100.0 
99i9 
99.7 
98.9 
97.1 
93.1 
85. 
7^.4 
59^a 
42.3 
.2<>.4 
14.0 
** 6*2 
2.2' 
0.6 
0.1 
^ 0.0 
0.0 



100.0 
100.0 
100.0 

106.0 

100.0 
100.0, 
100.0 
100.0 
99.9 
99.5 
98.6 
96.3 
91.6 
83.7 
72^2 
57.8 
.42.2 
27.8 
16.3 
6.4 
3.7 
1.4 
.0.5 
0.1 
-0>.0 
O.p 

o;o 

0.0^ 
0.0 
0.0 
0.0 



100.0 100.0 
100.0 96.8 



100^6 

99.9 
99.4 
9T.8 
93. P 
86;0 
73.6 
57.7 
40.7 
25.6 
K.2 
6.9 
2i9 
l.l 
0.3 
0.1 
0.0 

• o.d 

0.0 
0.0 

6^0 

0.0 ' 

0.0 

0.0 

0.0. 

0*0 

0;0 

0.0 

o;o ^ 



83.8 
60.4 
3^5.0 
16.0 
5.8 
1.7 
0.4 
0.1 
' .0. 0 
0«0 
^OiO 
VO.O 

<i.o 

0.0 
0.0 
0.0 
0.0^ 
0.0 
0.0 
0;0 

o.o 

0.0 
0.0 

0.6 
o.n 

0.0- 
0;0 
0.0 
0.0 : 
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\ PASS-FA^ COI^l 



CONTfNi;rNCY TA^*Lfc 
»NAM6 OF' 11 ST : 



0 



dRlGmL UAIA : ADMINISTRATIS DATE = 07/01/77 NUMBER 01^ EXAMINEES = 327A .NUMBER OF ITEMS = 30 

TyiS TABjLf IS' FOR A TEST OF 60 ITEMS , * o . 

TRUE-SCOkt CUTH'F^ : 0*f.f>67 - . ' / ^ , ^ \ . 

MINIMUM PARSING SCOkt JS' AO OUT flF 60^ ITEMS -PERCENT I . ' , 

SHOULD SHOULD . ' ^ 

PASS FAIL , ' ' ^ . * \ 

I ~ I I • • . ■ ' ; ^ : 

WILL PASS * I 57. 6t | fl.:^:: t ' 
\ ^ I . * , 

WILL FAIL I 7*1X |^ 27.0^ | ^ . ' ' ^ 



DECISION-MAKING EFFICIENCY : ^ ' ' * . > 



EXPECTATIONS (Z-Z*) SIGNCX-X^H 

DECISION-MAKING EFFICIENCY IS DEFINED AS ^ , 

^ EXPkCCTATlONCIZ-2*) SIGNCZ-Z^U 



WHE^E Z» IS TRUE-SCORE CUT-^FF OF 0^67 

Xi» IS OBSERVEO-SGORE CUT-OFF OF 40 > ' 
^ (MINIMUM PASSING SCORE MINUS 0N6-HALF ITEM) 



ERIC 
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COWOlTIdNAL TRUt-SCORt DI STR 1 bUT lUMS FUR PASSING AND FAILING GROUPS 

NAME OF Tt ST : , ^ * ' ^ ^ ' , ^ • 

ORIGINAL DAIA : ADMIf4lSTRAT IHN DATE = 07/01/77 NUMBER OF EXAMINfcES = ' 3274 



NUMBER OF ITEMS = 



THESt OISTRlBUTKiNS ARt FOlt 


A TEST OF 


60 ITEMS 






NUMBER 


TRUE- 


-SCORt . 


PASb ING 


FAI LING 


(PROPORT lUN-CdRRECT ) 


GROUP 


GR UUP 


C.95- 




0. 


0. 


.0.90r 


•0;9'i 


20. 


0. 


0.8!:- 




l^5. 


0. 


0.60- 




311. 


' 0. 


0.75- 


o.eu 




9. 


, 0.70- 


o.7r» 


. 603. 


60. 




0,70 


399. . , 


241. 


0.60- 


,0.6S 


^42. 




0.55-^ 


0.60 ^ • ' 


25v 


^.53. 


-0.50- 


0.55 


2. 


131. 


0.45- 


0.50 




5>1. 


0.40- 


0 .45 . 


'0. 


15. , 


K 0.3S- 


O.iiJ 


'0. 




0.30- 


0.35 


"0. 




0.25- 


0.30 * 


0. 




0.20- 


0.25 


0. 




' 0.15- 


0.20 


• 0. 


< 1: 


0.10- 


O.l*^ . 


0. > 




0.05- 


C.lO ^ ^' 






0.0 .r- 


0.05 






f 








' TRUE-SCORE > o 






MEAN 


0^74 


O.M ^ 


S.O. " V 


6.067 a 


0.074 



A MINIMUM PASSfNG ^ICOftEToF , 40 IT EMS CORR ECT 



PERCENT 
PASSING - FAILING 

GROUP GROUP 

* 'O.O 0.0. 

0.9 0.0^ 

5.3 Q.O 

1.4.4 0.0 

25.0 0.6 

.28.0 7.2 

18.5 21.6 

6.6 29.4 

1.2 2^.6 

0.1 11.7 

0.0 4.*? 
0.0 _ 1#A 

0.0 0*3 
0.0 . ^ 0.1 

o;o 0.1^- 

. 0.0 0.1' 

•0.0 0.1 

o-.o 0.1 • 

0.0 ? 0.1 



0.0 



0.1 



CUMULATIVE 
PASSING 
GROUP 
100.0 

. toowo 

99.1 
93.7 
. 79.3 
54.4 
26. A 
7*9 
1*3 
0.1 
' 0.0 
0.0 
0.0 
0.0 
0*0 
0.0 
0.0 
.0.0 
0.0 
0.0 



PERCENT 
FAILING 
• GROUP 
100.6 
100.0 

' idowo 

100.0 
100.0 
99.2 
92.0 
70.4 
41.0 
16.4 
6.7 
2.2 
O.R 
. 0.5 
0.4 . 
0.3 
0.3 
0.2 
0.1 . 
O.l 



A. Livingston 
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CmtilllUNLl l.«UF-SCe)Pt OISTlCYi?MTl<iN*FbR SINGt>e OBSERVeP-SCURF LEVEL 
NArtE OF TEjST : . - "^^ * . X 



ORIGINAL I API^INISTR'aTION f^ATt ^^7/01/77 ' 'NUMBER OF EXAMINEES' = 327^ 



NUMBEfi Of' ITEMS = 30 



IS DISTKIPMTION IS FOR A TFST OF 60 ITEMS 



\ 



FOR fXAMlNFFS WITH OBStRVeO-^SCORF ' 40 
' ^ TRUE^SCORt ^- 'NUflEER 

(PROPO^^TION-COKRECT ) • 
0,9t>- l.OO • 

0.90 



0,90- 
0 
0 

0.7&- 



85 

6a 



0.70- 0.^5 
0.65- 0.70 



0.60- 



0.65 
0.6O 



0.50- 0.55 



0.40- 



0.50 
0.45 



0.35- .0.40 

0.30- 0.35 
• 0.25- 0.30 

0.20- 0.?5 
^0.15- 0.?0 
^0.10-^.15 

0.05- O.IO • 

TRUB^SCORf 
/MEAN 

^ /s.o. 

4 



0.67 
0.048 



UUT OF 60 
PERCENT 



ITEMS^ { 67 PERCENT ^ 
CUMULATIVE 







PERCENT 


0. 


0.0 > 


100.0. 


0*. 


0.0 


100. 0 


0. 


0*0 


100.0' 


0. 


0.2 • 


• 10&,0 


9. 


5.0 ' 


99. 9 


46. 


25.2 


*>. 94.7 


72. 


39.5 


69.5 


43. 


23.5 


30.0 


11. 


5.8. 


6.5 


1. 


^.6 


' > 0.7 * 


0. 


0.0 


^ 0.0 ^ 


0. 


* 0.0 


0.0 


. 0. 


0.0 


0.0 


0. 


0.0 


Q.Q 


0. 


0.0 _ 


o\o 


^ 0. 


0.0 


0^0 


0. 


0.0 


0.0 


0. • • 


0.0 


0.0 


0* 


vJ 0.0 


0.0 


0. 


' e.o 


0.0 




Handout: page 8 



i \ 

V 



•PRtDICUh CnMClTlOfJAl QBSEnVF/D^'scORr.dlSTRlBUTlUNS FOR SEL^CTtO TRUE-SCORE LCVELs' 
NAMt OF 7f ST ' ' # • * • . 



ORIGINAL, DATA : AOMINISTKAll ON DATf i=r qV/0l/77» 
THESl DISTRIBUTIONS AP.E.>nK A.JEST fiF 60 'ITEMS 

NUN-CUMULATIVE OI'STRiaUTlONS 
T RUE-SCPkF J PROPPRT inN-Ct)RRfcCT > 
•10 ^30 \ .bO ^ ,70 .90' 



OBStRVEO 
J»8 - 
SfC - 
52 - 
49 - 
^ -6 r 
43 - 
40 - 
37 - 
34 - 
31 - 
?ft - 

22 

19 - 
16^ 
131- 
10 - 
7*- 
4 r- 
I - 



-SCORE 

4 ft. 
45 ' 

^? 
39 
36 
33 
30 

27. ^ 
24 

IB 
15 
12 
9 
6 
3 
0 



0.0 
0.0 
0.0 
0«0 
0.0 
' 0.0 
0.0 
0.0 

o.o- 
;.o*o 

0.0 
0.0 
0.0 
0.0 
0.3 
5.6 
^3.3 
49.0 
11.8 
0.1 



0% 

0.0 
0.0 
0.0 
0.0 

o.o 

0.0 

0.6 

0.0 
C.2 

2.4 

ll.A 

2^.0 
34^0, 
If^.? 

4«:i 

0.3 
-T.O 
0.0 
0*0 



0.0 
0.0 
0.0 
0.0 
0.0 
0.0- 
0.4 
^ 3.1 
13.0 
?8.0 
31.2 
18.1 
5.4 
G.B 
O.I 
0.0 
0.0 

,"o.o 

0.0 
0.0 
0.0 



0.^ 
0.0 
0.1» 

?.o 

12.2 
30.2 
33.^ 
1T.3 
4.4 
^1 0.6 
0.0 
0.0 
O.-O 
0.10 
0.0 
. O.O 
0.0 
0.0 
^ 0.0 
0.0 
0.0 



4.1 

' 3B,5 
44.9 
11.6 
0.9 
0.0 

o.a 

0.0 

0.0 

* 0.0 
0.0 
0.0 
0.0 
0.0 , 
0.0 
0.0 

q;6 

0.0 
0.0 
0.0 
.0.0 



NUMBER OF EXAMINEES = 3274 ' 

CUMULATIVE. OISTRtei^TIONS 
TRUE-SCORE (PROphRT ION-CORRECT >. 



NUMBER UF ITEMS = 30 



•10 .30 .50 . .70 



100.0 
100.0 
100.0 
* 100. 0 
100. C 
100.0 
100.0 
lOO.O 
100.0 
lt)0.0 
'lOD.O 
10^*0 
100.0 
100 .0 
100.0 
100.0 
99.7 
94.2 
60«8 
11.9 
,0.1 



100.0 
100^0 
100.0 
100.0 
100*0 
100.0 
100.0 
100.0 
100.0 
100.0 
100. 0 
99.7 
97.4 
65.6 
56.6 
22.6 
4.4 
0.3, 
0.0 
0.0 
0.0 



100.0 
100.0 
100.0 
lOOwO 

looi'o 

100«0 

iLpo.o 

99.6 
96.5 
83.5 
55.5 
24.4 
' 6.3 
0.9 
0.1 ' 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 



lOO.O: 
100«0 
100.0 
99.9 
97.9"^ 
85.6 
55.4 
22.2 
5.0 
0.6 
0.0 
0.0 

6.0 

0.0 

^ o.e 

0.0 

0.6 

0 

0. 
0. 
0. 



.0 
.0 



.90 

Uxo.o 

95.9 
57.4 
12.5 
' 0.9 
0^0 
0.0 
0.0 
0.0 
0.0 
0.0 

o«o. 

0.0 

0.0 ^ 
^ 0.0 I 

0.0 

0.0 
.f.0.0 
• 0.0 

0.0 
.0.0 



17 




.A 



S. A. Livingston 

15-7^8 



Handout: -page 9 



CONOITIONAL tRl«-SCOR£ 01 SIR I BUT IONS FOR PASSING AND FAILIKO GROUPS ANO**TRuk-SCORE OI^TRIWHON 
7 • f4AME OF ' TEST ^ * ■* * » ^ , ^ ■ / 

$2l2i^iiV*»?5I^i.iP"^^^^^''^^^^ O'^^ • 07/01/77 NUU8ER OF EXAMINEES - ^5274 NUMBER W IfTEUS - 30 
THESE DISTRIBUTION ARE FOR A TEST OF 00 ITEMS, WITH A MJNIMUM PASSING SCORE ^ 40 ITEMS CORRECT . 



< 
Li 

o 

tc 
Li 



. FAILINC 

PASSING 

TOTAL 




0,0O 



ERIC 



0?IO 



~i r 

0.20 .0*30 



0,50 \ o.eo V 

TR^-SCORE • 



0.70 



r 



O.eo 



18 \ 



0,9Q • I.OO 
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